The Camaleon Web Wrapper Engine

The web is rapidly becoming the universal repository of information. A major challenge is the ability to support the effective flow of information among the sources and services on the web and their interconnection with legacy systems that were designed to operate with traditional relational databases. This paper describes a technology and infrastructure to address these needs, based on the design of a web wrapper engine called Caméléon. Caméléon extracts data from web pages using declarative specification files that define extraction rules. Caméléon is based on the relational model and designed to work as a relational front-end to web sources. ODBC drivers can be used to send SQL queries to Caméléon. Query results by Caméléon are presented in either XML or HTML table formats. Users can also easily call Caméléon from other applications (e.g. Microsoft Excel by using Caméléon web query file (Caméléon.iqy)). Unlike its predecessor, Grenouille, Caméléon lets users segment web pages and define independent extraction patterns for each attribute. The HTTPClient package used in Caméléon supports both GET and POST methods and is able to deal with authentication, redirection, and cookie issues when connecting to web pages.